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Stochastic Averaging in Discrete Time and Its 
Applications to Extremum Seeking 

Shu-Jun Liu and Miroslav Krstic 


Abstract 

We investigate stochastic averaging theory for locally Lipschitz discrete-time nonlinear systems with stochastic 
perturbation and its applications to convergence analysis of discrete-time stochastic extremum seeking algorithms. 
Firstly, by defining two average systems (one is continuous time, the other is discrete time), we develop discrete¬ 
time stochastic averaging theorem for locally Lipschitz nonlinear systems with stochastic perturbation. Our results 
only need some simple and applicable conditions, which are easy to verify, and remove a significant restriction 
present in existing results: global Lipschitzness of the nonlinear vector field. Secondly, we provide a discrete-time 
stochastic extremum seeking algorithm for a static map, in which measurement noise is considered and an ergodic 
discrete-time stochastic process is used as the excitation signal. Finally, for discrete-time nonlinear dynamical 
systems, in which the output equilibrium map has an extremum, we present a discrete-time stochastic extremum 
seeking scheme and, with a singular perturbation reduction, we prove the stability of the reduced system. Compared 
with classical stochastic approximation methods, while the convergence that we prove is in a weaker sense, the 
conditions of the algorithm are easy to verify and no requirements (e.g., boundedness) are imposed on the algorithm 
itself. 


Index Terms 

Stochastic averaging, extremum seeking, stochastic perturbation 


I. Introduction 

The averaging method is a powerful and elegant asymptotic analysis technique for nonlinear time- 
varying dynamical systems. Its basic idea is to approximate the original system (time-varying and periodic 
or almost periodic, or randomly perturbed) by a simpler (average) system (time-invariant, deterministic) 
or some approximating diffusion system (a stochastic system simpler than the original one). Averaging 
method has received intensive interests in the analysis of nonlinear dynamical systems (D5]|, @, dTJ, ESI , 
ESI, lf35t |[9l , E71 . E2S), adaptive control or adaptive algorithms ([[24]], ESI), and optimization methods 

(0, 0, m, ED). 

Extremum seeking is a non-model based real-time optimization tool and also a method of adaptive 
control. Since the first proof of the convergence of extremum seeking IfTTl . the research on extremum 
seeking has triggered considerable interest in the theoretical control community ( ll34l . |[6l , [[33l . OD ¬ 
ES!, ECU, l!T6ll ) and in applied communities ( EDI . ED . ESI)- According the choice of probing signals, 
the research on the extremum seeking method can be simply classified into two types: deterministic ES 
method (0, flTJ, EU, ||3D , ESI, Il32l ) and stochastic ES method ( 081 , [IT51 , DUS). In the deterministic 
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ES, periodic (sinusoidal) excitation signals are primarily used to probe the nonlinearity and estimate its 
gradient. The random trajectory is preferable in some source tasks where the orthogonality requirements 
on the elements of the periodic perturbation vector pose an implementation challenge for high dimensional 
systems. Thus there is merit in investigating the use of stochastic perturbations within the ES architecture 

( 03 ). 

In fl5ll . we establish a framework of continuous-time stochastic extremum seeking algorithms by 
developing general stochastic averaging theory in continuous time. However, there exists a need to consider 
stochastic extremum seeking in discrete time due to computer implementation. Discrete-time extremum 
seeking with stochastic perturbation is investigated without measurement noise in |[T8Tl , in which the 
convergence of the algorithm involves strong restrictions on the iteration process. In 071 and [|32l . discrete¬ 
time extremum seeking with sinusoidal perturbation is studied with measurement noise considered and 
the proof of the convergence is based on the classical idea of stochastic approximation method, in which 
the boundedness of iteration sequence is assumed to guarantee the convergence of the algorithm. 

In this paper, we investigate stochastic averaging for a class of discrete-time locally Lipschitz nonlinear 
systems with stochastic perturbation and then present discrete-time stochastic extremum seeking algorithm. 
In the first part, we develop general discrete-time stochastic averaging theory by the following four steps: 
(i) we introduce two average systems: one is discrete-time average system, the other is continuous-time 
average system; (ii) by a time-scale transformation, we establish a general stochastic averaging principle 
between the continuous-time average system and the original system in the continuous-time form; (iii) 
With the help of the continuous-time average system, we establish stochastic averaging principle between 
the discrete-time average system and the original system; (iv) we establish some related stability theorems 
for the original system. To the best of our knowledge, this is the first work about discrete-time stochastic 
averaging for locally Lipschitz nonlinear systems. 

In the second part, we investigate general discrete-time stochastic extremum seeking with stochastic 
perturbation and measurement noise. We supply discrete-time stochastic extremum seeking algorithm for a 
static map and analyze stochastic extremum seeking scheme for nonlinear dynamical systems with output 
equilibrium map. With the help of our developed discrete-time stochastic averaging theory, we prove 
the convergence of the algorithms. Unlike in the continuous-time case fl5ll . in this work we consider 
the measurement noise, which is assumed to be bounded and ergodic stochastic process. In the classical 
stochastic approximation method, boundedness condition or other restrictions are imposed on the iteration 
algorithm itself to achieve the convergence with probability one. In our stochastic discrete-time algorithm, 


3 


the convergence condition is only imposed on the cost function or considered systems and is easy to 
verify, but as a consequence, we obtain a weaker form of convergence. Different from [|T6l in which 
unified frameworks are proposed for extremum seeking of general nonlinear plants based on a sampled- 
data control law, we use the averaging method to analyze the stability of estimation error systems and 
avoid to verify the decaying property with a JCJ 2 ? function of iteration sequence (the output sequence of 
extremum seeking controller), but we need justify the stability of average system. 

The remainder of the paper is organized as follows. In Section |TT| we give problem formulation of 


discrete-time stochastic averaging. In Section III we establish our discrete-time stochastic averaging 


theorems, whose proofs are given in the Appendix. In Section [TV] we present stochastic extremum seeking 
algorithms for a static map. In Section [V] we give stochastic extremum seeking scheme for dynamical 
systems and its stability analysis. In Section [VI] we offer some concluding remarks. 

Notation: Co(R”) denotes the family of all continuous functions on M" with compact supports, [jc] 
denotes the largest integer less than x. 

II. Problem formulation of discrete-time stochastic averaging 
Consider system 


X k+i =X k + £f(X k ,Y k+1 ), k = 0,1,2,..., (1) 

where X k G W 1 is the state, { Y k } C R m is a stochastic perturbation sequence defined on a complete 
probability space where Q. is the sample space, & is the <7-field, and P is the probability 

measure. Let Sy C R” 7 be the living space of the perturbation process. £ G (0,£o) is a small parameter for 
some fixed positive constant £q. 

The following assumptions will be considered. 

Assumption 1: The vector field f(x,y) is a continuous function of (x,y), and for any x G R", it is a 
bounded function of y. Further it satisfies the locally Lipschitz condition in x G R' 7 uniformly in y & Sy, 
i.e., for any compact subset D C R", there is a constant ku such that for all xi,x 2 G D and all y G Sy, 


\f(pci,y)~f(x 2 ,y)\ <k D \xi~x 2 \. 

Assumption 2: The perturbation process {Y k } is ergodic with invariant distribution p. 
Under Assumption [2j we define two classes of average system of system (|T|i) as follows: 

Discrete average system: X k+l — X k + £f(X k ), k — 0,1,..., 

dX c (t) 

Continuous average system: —-— = f(X c (t)), t > 0, 


( 2 ) 

(3) 
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By Assumption [TJ f(x,y ) is bounded with respect to y, thus y —* f(x,y ) is /i-integrable, so / is well 


defined. Here the definition of discrete average system is different from that in ll29l . where the average 
vector field is defined by f(x) = E f(x. Y k+ \) (there, the perturbation process {f^+i} is assumed to be 
strict stationary). In this paper, we consider ergodic process as perturbation. It is easy to find discrete-time 
ergodic processes, e.g., 

• i.i.d random variable sequence; 

• finite state irreducible and aperiodic Markov process; 

• {Y{, i = 0,1,...,} where {Y t ,t> 0} is an Ornstein-Uhlenbeck (OU) process. In fact, for any continuous¬ 
time ergodic process {Y t . t > 0}, the subsequence (I), i = 0,1,...,} is a discrete-time ergodic process. 

For discrete average system ©, the solution can be obtained by iteration, thus the existence and 
uniqueness of the solution can be guaranteed by the local Lipschitzness of nonlinear vector field. For 
continuous average system ©, f(x) is easy to be verified to be locally Lipschitz since f(x,y ) is locally 
Lipschitz in x. Thus, there exists a unique solution on [0, (Too), where o, XJ is the explosion time. Thus we 
only need the following assumption. 

Assumption 3: The continuous average system ([5]) has a solution on [0, +°°). 

By©, we have 


k 


X k+l =X 0 + e'£f(X i ,Y i +i). 


(5) 


i =0 


We introduce a new time t k — ek. Denote m(t ) = ma x{k :t k <t} and define X (/) as a piecewise constant 
version of X k , i.e., 


X(t) =X k , as tk < t < t k+ 1 , 


( 6 ) 


and Y(t) as a piecewise constant version of Y n , i.e., 


Y{t) =Y k , as t k <t <t k+ 1 . 


(7) 


Then we can write © in the following form: 


m(t) 


X(t)=X 0 + eY,f(X k -hY k ) 


k =\ 


( 8 ) 
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or as the continuous-time version 

X(t)=X o+ f f(X(s),Y(e + s))ds- f f(X(s),Y(e + s ))d S . (9) 

JO Jt m (r) 

Similarly, we can write the discrete average system ([2]) in the following continuous-time version 

X d (t)=X 0 + ff(x\s))ds^ f f(X d {s))ds, (10) 

and write the continuous average system ([3]) by 

X c (t)=X. o+ f f(X c (s))ds, (11) 

Jo 

where X d (/) is a piecewise constant version of X d , i.e., X d (t) = X d . as t k < t < \. We now rewrite the 

continuous-time version ([9]) of the original system (|Tj) as two forms: 

X(t)=X o+ f f(X(s))ds- f f(X(s))ds+R^(t,X(-),Y(e + -)), (12) 

JO Jt m (t) 

X(t)=X o+ f f(X(s))ds + R^(t,X(-),Y(e + -)), (13) 

Jo 

where 

i?«(r,Z(-),F(e + -)) = (f(X(s),Y(e + s))-f(X(s)))ds, 

R ( ~ 2 \t,X(-),Y(e + -)) = J‘ (f(X(s),Y(e + s))-f(X(s)))ds 

~ [ f(X(s),Y(e + s))ds. 

Hence we consider system ( [12]) as a random perturbation of the continuous-time version pO] ) of discrete 
average system ([2]) and consider system ( [13] ) as a random perturbation of the continuous average system 

(HD). 

To study the solution property of the original system #■ we develop discrete-time stochastic averaging 
principle, i.e., using average systems ([2]) or (|3]) to approximate the original system ([!]). 

Different to some existing discrete-time stochastic averaging results Il29l . we consider averaging results 
under some weaker conditions: (a) the nonlinear vector field is locally Lipschitz; (b) the perturbation 
process is ergodic without other limitations. Under these weaker conditions, we can obtain weaker 
approximation results. The main idea is as follows. First, we use the solution of continuous average 
system ([3]) to approximate the solution of continuous-time version ([9]) of the original system (jT|), and then 
prove that the solution of the continuous-time version ( fTO] ) of the discrete average system Q and the 
solution of the continuous average system ([3]) are close to each other as small parameter £ is sufficiently 
small. Thus we can obtain that the discrete average system ([2]) can approximate the original system ([[]) 
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by transforming the continuous-time scale back to discrete-time scale. The main idea can be simplified 
as Fig. [T] (| • | —>• 0 means the convergence in some probability sense). 


Step 1. 
Step 2. 
Step 3. 
Step 4. 


X c (0-*(0|^0, asc^O 
X d (O-X c (O|->0, as £ ->0 
X d (t)-X(O|^0, as s —» 0 
X d -Xj->0, as £• -> 0 


Fig. 1. Main idea for the proof of discrete-time stochastic averaging. 


Remark 2.1: Our developed averaging theory is also applicable to the following systems 


X k+i = X k + £ (. f(X k , Y k+l ) + W k+1 ), k = 0,1,2,..., (14) 

where {114} C W 1 is bounded with a bound M and ergodic stochastic sequence, which is independent of 
the perturbation sequence {Y k }. 

Take a function g G Co(M) such that g(x) = 1, for any v G Bm{ 0) = {x G M' ! : \x\ < M} and denote 
F(X k ,Z k+ i) = f(X k ,Y k+l ) + g(W k+ 1 ). Then we obtain the following system 


X k +i =X k + £F(X k .Z k+ \). k = 0,1,2,.... 


(15) 


Since {W k G M”} and {Y k } are independent and ergodic, we can obtain the combination process Z k = 
{(Y k , Wjf) T } is also ergodic. It is easy to check that the new system (15) satisfies Assumption [TJ Thus 
we know system ([T4]) is included into our considered system (jTj). 


III. Statements of General Results on Discrete-time Stochastic Averaging 

Denote X(t), X c (t), X d (t), as the solutions of system ([8])(the same as (|9|), continuous average system 
CD- and continuous-time version ( flO] ) of discrete average system ([2]), respectively. To avoid too complex 
mathematical symbols, we do not manifest X(t) and X c (t) to be dependent on small parameter £. 

We have the following results: 

Lemma 1: Consider continuous-time version ([9]) of the original system under Assumptions [T] |2] and [3] 
Then for any T > 0, 


lim sup \X(t) — A c (f)| = 0 a.s. 

£ ^°0<t<T 


Proof: See Appendix |A} 


(16) 
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Lemma 2: Consider continuous-time version ([9]) of the original system under Assumptions [T] and |2j 
Then for any T > 0, 

lim sup |X(r) -X d (t)| = 0 a.s. (17) 

e ->°o <t<T 

Proof: See Appendix [Bj ■ 

Lemmas [I] and [2] supply finite-time approximation results in the sense of almost sure convergence. In 
ffHL for any 8 > 0, lim e _> 0 P(sup 0 < f < r \X(t) -X c (t)| >5) = 0 or lim e ^ 0 Psup 0 < ; < r |X(t) -X c (t)| =0 
can be achieved, but the nonlinear system is required to be globally Lipschitz. In this work, we present 
approximation result for locally Lipschitz systems. 


A. Approximation and stability results with continuous average system 

In this subsection, we present approximation results to the original system and the related stability 
results with the help of continuous average system. First, we extend the finite-time approximation result 
in Lemma |T| to arbitrarily long time intervals. 

Theorem 3: Consider system ([9]) under Assumptions |T] [2] and [3] Then 

(i) for any 8 > 0, 

lim infjr > 0 : |X(f) — X c (t)| > 5} = +°° a.s. (18) 

£->0 

(ii) there exists a function T(e) : (0,£o) —> N such that for any 5 > 0, 

lim pi sup |A(t)-A c (t)| > 8 \ -0 (19) 

e ^° [o<f<r(£) J 

with 


lim Tie) = +oo. 

£—^-0 


( 20 ) 


Proof: See Appendix [C] for the result ( p~8| ), and Appendix |D] for the result ( [T9] ). ■ 

Similar to the continuous case lfT5l . although it is not assumed that the original system (|T|) or its 
continuous-time version (|9j) has an equilibrium, average systems may have stable equilibria. We consider 
continuous-time version S of the original system as a perturbation of continuous average system ([3]), 
and analyze weak stability properties by studying equilibrium stability of continuous average system ([3]). 

Theorem 4: Consider continuous-time version ([9]) of the original system under Assumptions [T[ [2] and [3] 
Then if the equilibrium X c {t ) = 0 of continuous average system ([3]) is exponentially stable, then it is weakly 
exponentially stable under random perturbation R^\t, A(-),T(e-t- •)), i.e., there exist constants r > 0, c > 0 




7 > 0 and a function T (e): (0, Co) —>■ N such that for any initial condition X c (0) = Xo = x G {x e M" | |Jc| < r}, 
and any 8 > 0, the solution of system ([9]) satisfies 

lim inf {t > 0 : |X(t)| > c\x\e~^ + 5} = +°° a.s. (21) 

e-K) l j 

and 

lim P{\X(t)\ <c\x\e~V + §y t e [0,r(e)]| = 1 (22) 

£->0 L ’ 

with lim e ^o^( £ ) — +°°- 

Proof: See Appendix [EJ ■ 

Remark 3.1: By analyzing the weak stability of continuous average system, we obtain the solution 
property of system ([9]). Here the term “weakly” is used because the properties in question involve lim e ^o 
and are defined through the first exist time from a set. In IflOl . stability concepts that are similarly defined 
under random perturbations are introduced for a nonlinear system perturbed by a stochastic process. 


B. Approximation and stability results with continuous-time version of discrete average system 

Similarly, we can extend the finite-time approximation result in Lemma [2] to arbitrarily long time 
intervals. 

Theorem 5: Consider continuous-time version ([9]) of the original system (|T[) under Assumptions [T] and 
121 Then 

(i) for any 8 > 0, 


lim infjt > 0 : |A(t) — a (r)| > 5} = +°° a.s.; 
£->0 


(23) 


(ii) there exists a function T(e) : (0,£q) N such that for any 8 > 0, 


I 


lim Pi sup |A(t) — A d (t)| > 8 /> = 0 

e ^° [o<;<r(£) 


(24) 


with lim e ^o2"( £ ) = +°°- 

Proof: See Appendix [FJ ■ 

Similar as the last subsection, if the continuous-time version ( [T0| ) of discrete average system ([2]) has 
some stability property, we can obtain the solution property of the continuous-time version (|9]) of the 
original system ([[]). 

Theorem 6: Consider continuous-time version ([9]) of the original system (|T) under Assumptions |T| 
and [2] Then if for any e e (0,£o), the equilibrium A d (/) = 0 of continuous-time version (10) of discrete 
average system Q is exponentially stable, then it is weakly exponentially stable under random perturbation 
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R (1 )(t,X(-), Y(£ + •)), i.e., there exist constants r £ > 0, c E > 0, y e > 0, and a function T(e) : (0, So) —>■ N 
such that for any initial condition X d = Xo = x G {x € W 1 : \x\ < r £ }, and any S > 0, the solution of system 
® satisfies 

lim infjt > 0 : |X(f)| > c e |x|e _%f + 5} = +°° a.s. (25) 

e->0 L 1 

and 


lim P{|X(t)| <c £ \x\e~ rEt + 8,Vt e [0,r(e)]| = 1 
e^0 L J 


(26) 


with lim e ^o2"(e) = +°°. If the equilibrium X d (t) = 0 of continuous-time version (10) of discrete average 
system ([ 2 ]) is exponentially stable uniformly w.r.t. £ 6 ( 0 , £o), then the above constants r e ,c £ > 0 and y e 
can be taken independent of £. 

Proof: See Appendix [Gj ■ 

Remark 3.2: Notice that continuous average system ([3]) is independent of small parameter £, but discrete 
average system ([ 2 ]) or its continuous-time version (fT0|) is dependent on £. The stability results (25)-(26) 


are different from (|2T|)-(|22|), but in the discrete-time scale, the results are the same in essence, which can 
be seen in the next subsection. 


C. Approximation and stability results in the discrete-time scale 


In the last two subsections, we consider the approximation and stability analysis in the continuous time 
scale. Since the original system is in the discrete time scale, now we consider averaging results in the 
discrete time scale. 

Let {X^} and {X d } be the solutions of the original system (|Tj), discrete average system ([Ij), respectively. 
Rewrite system ([[]) as 

X k+l =X k + £f(X k ) +R^(X k ,Y k+l ), k = 0,1,2,..., (27) 


where R^(X k ,Y k+ 1 ) = e(f(X k ,Y k+ 1 ) — f(X k )). Hence we can consider system (27) (i.e. system (JlJ)) as 


a random perturbation of discrete average system (|2]). Thus we can analyze the weak stability of (|2]) to 
obtain the solution property of ( [27] ) (i.e. system ([!])). 

We have the following approximation results. 

Lemma 7: Consider system ([[]) under Assumptions |T| and [2] Then for any N e N, 

lim sup \X k — X A d |=0 a.s. (28) 

£ ^°0<yt<[W/e] 


Proof: See Appendix [H} 











Theorem 8: Consider system (JTJ) under Assumptions [T] and |2j Then we have 
(i) for any 8 > 0 , 
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lim infjk e N : —X, d | > 5} = +°° a.s.; 

e ->0 

(ii) for any 8 > 0 and any N G N, 


(29) 


lim P 

£->0 


\ Sup \X k 

[o</K[iV/£] 


X d | > 8 


0 . 


(30) 


Proof: See Appendix |Tj ■ 

To the best of our knowledge, this is the first approximation result of stochastic averaging in discrete 
time for locally Lipschitz systems. Il29l developed discrete-time averaging theory, which mainly focuses 
on globally Lipschitz systems. 

About the solution property of the original system ([T]) by analyzing the stability of discrete average 
systems Q, we have the following results. 

Theorem 9: Consider system ([I]) under Assumptions [I] and [2] Then if for any £ e (0,£o), the equilibrium 
A d = 0 of the discrete average system (J 2 J) is exponentially stable, then it is weakly exponentially stable 
under random perturbation R^(X k ,Y k+ i), i.e., there exist constants r £ > 0 , c £ > 0 and 0 < y e < 1 such that 
for any initial condition Xq = x G {x G M" : \x\ < r £ }, and any 8 > 0, the solution of system (|T|) satisfies 

lim infjk e N : \X k \ > c e |x|(y e ) /c -|-5} = a.s. (31) 

£->0 l J 


and 


lim 1 P { \X k | < c e |x| {Yet + 5, Vk = 0,1,..., [N/e] ) - 1, (32) 

£->0 l J 

where N is any natural number. If the equilibrium A d = 0 of discrete average system (|2j) is exponentially 
stable uniformly w.r.t. £ e ( 0 , £ 0 ), then the above constants r £ ,c £ > 0 and y e can be taken independent of 
£. 

Proof: See Appendix [J] ■ 

Remark 3.3: Similar to continuous-time stochastic averaging in |[T5ll . if the discrete average system has 
other properties (boundedness, attractivity, stability and asymptotic stability), we can obtain corresponding 
(boundedness, attractivity, stability and asymptotic stability) results for the original system. 

Remark 3.4: The stability results in the above Theorems [4| [ 6 j and [9] are local, but if the equilibrium 
of corresponding average systems is globally stable (asymptotical stable or exponentially stable), then it 
is globally weakly stable (asymptotical stable or exponentially stable, respectively). 
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Remark 3.5: In analyzing the similar kind of system as (jT|), average method is different from ordinary 
differential equation (ODE) method and weak convergence method of stochastic approximation ( liTTl . 
GOl, lH2lO . In ODE method (gain coefficients are changing with iteration steps) and weak convergence 
method (constant gain), regression function or cost function (i.e., /(•) in system (|TJ)) acts as the nonlinear 
vector field of an ordinary differential equation, which is used to compare with the original system. In 
both methods, to analyze the convergence of the solution of system ([!]), there need some restrictions 
on both the growth rate of nonlinear vector fields and the algorithm itself (i.e., uniformly bounded), or 
the existence of some continuously differential function (Lyapunov function) satisfying some conditions. 
In some sense, these conditions (i.e. boundedness of iteration sequence) are not easy to verify. Average 
method is to use a new system (namely, average system) to approximate system <{T|) and thus makes it 
possible to obtain the solution property of system (|Tj). From Theorems [8] and [9| we can see that the 
approximation conditions are easy to verify. 


IV. Discrete-time Stochastic Extremum seeking Algorithm for Static Map 
Consider the quadratic function 

< P(X ) = (P* + -y(*-X*) 2 , 


(33) 


where x* E M, (p* E M, and (p" E M. are unknown. Any C 2 function (p(-) with an extremum at .* = x* and 


with (p V 0 can be locally approximated by (33). Without loss of generality, we assume that <p > 0. 


In this section, we design an algorithm to make \xk~ x*\ as small as possible, so that the output (p(xk) 
is driven to its minimum (p*. The only available information is the output y — (p (. 1 +) with measurement 
noise. 

Denote 4 as the k step estimate of the unknown optimal input x*. Design iteration algorithm as 


4+1 = 4 - £ sin(4 +1 )y fe+ i, k = 0,1 ,..., 


(34) 


where y^+i = <p(v) + V4 +1 is the measurement output, {is an ergodic stochastic process with invariant 
distribution /i and living space S v , and {14} is the measurement noise, which is assumed to be bounded 
with a bound M and ergodic with invariant distribution v and living space .SV. £ E (0, Co) is a positive small 
parameter for some constant £o > 0. The perturbation process {i+} is independent of the measurement 
noise process {V4}. 

Define Xk — 4 + asin(i+ + |), a > 0 and the estimation error 4 — 4 —4. Then we have 

11 

ty* + -y- (4 + ^sin(vfe+i)) 2 +14 + | 


4+1 = 4 — £sin(vjt + i) 


(35) 
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a sin(v t+1 ) esin(v t+1 ) 


Fig. 2. Discrete-time stochastic extremum seeking scheme for a static map. 


To analyze the solution property of the error system (J35J), we will use stochastic averaging theory developed 
in the last section. First, to calculate the average system, we choose the excitation process {v/J as a 

i - - y2 

sequence of i.i.d. random variable with invariant distribution fl(dy) — —TT dy. We assume that the 
measurement noise process {W/,} is any bounded ergodic process. 

By 0, we have 


Ave{shT(n. +i )}= / sin l (y)n(dy) = 0, i= 1,3, 
2/ 


Sy 

L 


Ave{sin-(v fc+ i)}= / sin 2 (y)/i{dy) = \~\e 2a ~ 


Ave{sin(v fc+ i)W fc+i }= 


' Sy x S \y 


sin(y).v/j(Jy) x v(dx) 


— sin(y)/i(Jy) x / xv(dx) — 0. 
J S v J S\y 


Thus, we obtain the average system of the error system (35) 

aq>\ l-e~ 2cyl ] 


,~ave _ ~ave e 
x k+1 — x k 


~ave 

x k 


= 1 — e 


a(p (1 — e 


-la 1 


(36) 

(37) 

(38) 


(39) 


Since (p" > 0, there exists £* = — Tl — " such that the average system (|39|) is globally exponentially 

a(p ( 1 — e~ 2a ~) 


stable for £ e (0, £*). 

Thus by Theorem [9j Remark 2.1 and Remark 3.4 for the discrete-time stochastic extremum seeking 
algorithm in Fig. [2] we have the following theorem. 


Theorem 10: Consider the static map ( |33| ) under the iteration algorithm ( |34| ). Then there exist constants 
c e > 0 and 0 < y t - < 1 such that for any initial condition jfo £ M and any 8 > 0, 

lim inf \k £ N : \x k \ > c £ |i 0 | iye) k + = +°° a.s. (40) 

e->0 f j 





























and 
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lim P 1**1 < c e |* 0 |{jef + 5,\/k = 0,1,..., [N/e] = 1. 

e -»0 


(41) 


These two results imply that the norm of the error vector jc* exponentially converges, both almost surely 
and in probability, to below an arbitrarily small residual value S over an arbitrarily long time interval, 
which tends to infinity as £ goes to zero. To quantify the output convergence to the extremum, for any 
£ > 0, define a stopping time 


Tg = inf jfc e N : |**| > c £ |x 0 | (,Ye) k + 8 


Then by d40b, we know that lim r| = +°°, a.s. and 

e->0 


|**| < Ce |*01 (7e) fc + Vk < T, 


£ * 


(42) 


Since y k +\ = (p(x* + jc* + asin(v*+i)) + W k +i and (p'(x*) — 0, we have 

y k+ \ - (p(x*) = ^ * (x* + asin(v* + i)) 2 

+ 0((x* + asin(v* + i)) 3 ) +W k+i . 


Thus by (42) and |W4+i| <M, it holds that 

b*+1 — ( P( x *) I ^ 0(a ~) + 0(8 2 ) +C E |x 0 1 2 (7e)" fc +M,\/k < z £ , 

for some positive constant C £ . Similarly, by ( |4T] ), 

fimp{|y fe+ i -<p(**)| < 0(a 2 ) + 0(8 2 )+C £ \x Q \ 2 (y £ ) lk , 
£->0 t 

+M, Vk = 0,l,...,[N/e]} = 1, 


(43) 


which implies that the output can exponentially approach to the extremum <p(x*) if a is chosen sufficiently 


small and the measurement noise can be ignored (M = 0). By (39), we can see that smaller £ is, slower 


the convergence rate of the average error system is. Thus, for this static map, the parameter £ is designed 
to consider the tradeoff of the convergence rate and convergence precision. 

Remark 4.1: As an optimization method, besides the different derivative estimation methods, there 
are some other differences between stochastic extremum seeking (SES) and stochastic approximation 
(SA)( [f30l . Il32l ). First, in the iteration, the gain coefficients in SA is often changing with the iteration 
step, but for SES, the gain coefficient is a small constant and denotes the amplitude of the excitation 
signal; Second, stochastic approximation may consider more kinds of measurement noise (i.e., martingale 
difference sequence, some kind of infinite correlated sequence), but here we assume the measurement noise 
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-the output y k 


-iteration sequence 

-extremum value <p* 


| - extremum point x* 
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Fig. 3. Discrete-time stochastic ES with independent Gaussian random variable sequence as the stochastic probing signal. 

as bounded ergodic stochastic sequence (the boundedness is to guarantee the existence of the integral in 
0 ; Third, to prove the convergence of the algorithm (P{lim^^x^ =jc*} = 1) , SA algorithm requires 
some restrictions on the cost function (regression function) or the iteration sequence, while the convergence 
conditions of SES algorithm are simple and easy to verify. 

Fig |3] displays the simulation results with (p* — l,(p" — 1,jc* = 1, in the static map (33) and a — 0.8,e = 


0.002 in the parameter update law (34) and initial condition xq — 5. The probing signal { v k } is taken as a 
sequence of i.i.d. gaussian random variables with distribution N( 0,2 2 ) and the measurement noise {114} 
is taken as a sequence of truncated i.i.d. gaussian random variables with distribution A(0,0.2 2 ). 

V. Discrete-time Stochastic Extremum Seeking for Dynamic Systems 
Consider a general nonlinear model 

x k- 1-1 = f( x kiU k ), 

y°k = h ( x k ), k = 0,1,2 ,..., 


where Xk G R" is the state, u k G R is the input, y9 G R is the nominal output, and / : 
h : R" —» R are smooth functions. Suppose that we know a smooth control law 

Uk = P(x k ,6) 

parameterized by a scalar parameter 0. Then the closed-loop system 

Xk +1 = f(xk,P(x k ,6)) 


X 


—y 


(44) 

(45) 
” and 


(46) 


(47) 


has equilibria parameterized by 0. We make the following assumptions about the closed-loop system. 
Assumption 4: There exists a smooth function /: R —> R' 7 such that 


f(x k , f5 (x k . 0)) =x k if and only if x k = 1(B). 


(48) 
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a sin(v M ) sin(v M ) 


Fig. 4. Discrete-time stochastic extremum seeking scheme for nonlinear dynamics 


Assumption 5: There exists 0* e M such that 


(hol)'(0*) = 0, 

(49) 

{hoi)"(6*) <0. 

(50) 


Thus, we assume that the output equilibrium map y = h(l(d)) has a local maximum at 6 — 9*. 

Our objective is to develop a feedback mechanism which makes the output equilibrium map y — 
( h(l{6 ))) as close as possible to the maximum y* — h{l{9*)) but without requiring the knowledge of 
either 0* or the functions h and /. The only available information is the output with measurement noise. 
As discrete-time stochastic extremum seeking scheme in Fig. |4j we choose the parameter update law 


6 k+1 = d k + £pZ k , (51) 

5fc+i = ^k-ewi^ k Aewi(y k+ x~^ k )sva(v k+ i), (52) 

Ck+ 1 = Ck - £W 2 C k + £W 2 yk+l, (53) 

yk+i=y° k + w k +i, (54) 


where p > 0,wi > 0,w 2 > 0,£ > 0 are design parameters, {v^} is assumed to be a sequence of i.i.d. Gaus- 

_ -r 2 

sian random variable with distribution p(dx) = ^-, Ka e la2 dx, and = (— M) VZ^AM is measurement 

*2 


noise, where {Z^.} is a sequence of i.i.d. Gaussian random variable with distribution v(dx) — ^ Jia e 2ar dx. 
We assume that the probing signal {vy} is independent of the measure noise process {}. It is easy to 
verify that {WjJ is a bounded and ergodic process with invariant distribution 


where q\ 
Define 


Vi(A) = v(A A (— M,M)) +qi +q 2 , for any ACM 


v([M,+°o)), if MeA _ I v((-oo,-M]), if -MeA 

0 , else ’ and qi ~ { 0, else 


(55) 


6 k = 6 k + a sin(vt+i). 


(56) 
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Then we obtain the closed-loop system as 

Xk+i = f(xk,P(x k ,d k + asm(v k+1 ))), (57) 

G k +i = Qk + ep%k, (58) 

4+t =^jt-ewi§fc + ewi(yJ + Wifc +1 -Cfc)sin(vjfc + i), (59) 

4+1 = Ck ~ £w 2 t,k + £w 2 {y° k + W*+i). (60) 

With the error variable 

e k = e k -e*, (6i) 

l k = &-h(l(0*)), (62) 

the closed-loop system is rewritten as 

x k +\ = f(x k ,P(x k ,d k + G*+asin(v k+ i)))i (63) 

G k +i = d k + ep^ k , (64) 

4 +1 =^-ewt^ + ewi (|/i(^)«/i(/(0*))*-4 + ^+i) sin(vjt +1 ), (65) 

4+t = 4-e^24 + e^ 2 (/r(^)-/r(/(0*))-MT fc+ i). (66) 


We employ a singular perturbation reduction, freeze x k in (63) at its quasi-steady state value as 


x k = 1(6* + 6 k + a sin(v*;+i)) 


and substitute it into (64)-(66), and then get the reduced system 


% + i = % + ep& 

5 +1 = ^ k -£w\^ k + £w\ ^g(G l k + asin(v k+ i))-Q + W k+ ^sm(v k+ i) : 

5+t = 5 - £^5 + £m; 2 (45 + asin(vjt +1 )) + W*+i). 
where g(d[ + a sin(v^+i)) = h (l(G* + 0^ + asin(vyt+i))) — h(l(6*)). With Assumption 0 we have 

40) = o, 

4(0) = (hol)'(G*)=0, 

g"(0) = (hoi)"(6*) < o. 


(67) 

( 68 ) 

(69) 

(70) 

(71) 

(72) 

(73) 







17 


Now we use our stochastic averaging theorems to analyze the reduced system (68>-(7()>. According to 


we obtain that the average system of (68>-(70) is 


^r,ave n r i ave i 

^k+ 1 “ t> k 

p r,ave p r,ave 

$k +1 ~^k 
?r,ave ?r,ave 

*k+ 1 _ ^k 


= e 


P^k 

^r.ave 


r,ave 


-wi^ a r + Is v ?( 0 r Ve + asin 0)) sin(y)ju(^) 
-w 2 C r ' ave +w 2 fs„ g (0/ r,ave + a sin (y))n(dy) 


(74) 


where we use the following facts: 

/ xVi(dx) — 0, / .vsin(y)Vi(J.r) x jd (dy) = 0. 

J iSjy J XiSy 

Now, we determine the average equilibrium (0 a,e , <^ a,e , C a,e ) which satisfies 

^ a ’ e = 0, 

—wi<^ a,e + w\ / g(0 a,e + asin(y)) sin(y)/i(dy) = 0, 
Js v 

—W 9 ^ a,e + W 2 [ g(B a,e + asin{y))id(dy) = 0 . 

./s„ 


We assume that 0 ae has the form 


0 a,e = b\a + b 2 d 2 + 0(a 3 


By © and ( |72] >, define 




(75) 

(76) 

(77) 

(78) 

(79) 


2 


3! 


(80) 
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Then substituting ( [79] ) and ( [80] ) into ( |77| ) and noticing that S v = R, we have 

/* —|—oo 


g(b\a + b 2 a" + 0(a i ) + asin(y)) sin{y) 


1 

e 2 ct- dy 


V'(0) 


{b\a4-b 2 cC + 0(o 3 ) + asin(y)) 


V2tzc 
2 


g w (0) 

3! 


( 7>](3 + & 2« 2 + <9(« 3 ) -t-asin(y))" 


,2 

e 2 (T 2 


+O((&ia-l-& 2 tf“ + 0 (tf 3 )+asin(y)) )1 sin(;y)^^^d;y 

\J2tic 


i —CO 

-/// 


V'(0) 


{2b\a 2 + 2 Z? 2« 3 + <9(c/ 4 )) sin 2 (y) 


f- ^ (3fc 2 fl 3 + (9(a 4 ) +a 3 sin 2 (y)) sin 2 (y) 


3! 

1 


x 


V2ne 


e la 1 dy + 0(a 4 


= O(a 4 ) + s"(0)b l [ l -- l -e- 1 ° 1 ^j a 2 

1 1 


b2g'\0) + b] 


+ 


g^(0)/3_l 

6 V 8 2 


2 2 e 


-2a 2 


2a 2 _|_ Ig-StT 2 


a 3 = 0, 


( 81 ) 


where the following facts are used: 

1 / , +°° 


V2tu7 J 

1 r+°° 

\/2kc J-oo 
1 /■+ o ° 


sin 


2k+l 


(y)e **dy = 0, k = 0,1,2,, 


sin 2 (y)e-idy='-- 1 - e - 2 ° 2 , 


3 1 -2a 2 , 1 ■-8ct 2 


\/2no ■ 


sin 4 (y)e 20 2 dy — - — -e “ a + -e 


8 2 


8 


(82) 


Comparing the coefficients of the powers of a on the right-hand and left-hand sides of (81), we have 


bi=0, 

h = 


g'"(0)(3-4e- 2a ‘+ e~ s<7 ~) 


24g // (0)(l — e~ 2cy2 ) 


(83) 

(84) 


and thus by ([79]), we have 


- ae q m 0 )(3-4e- 2(y +e~ S(y ) 2 . 3 , 

9 ' = - 3 A -- J -a 2 + 0(a 3 ). 

24<g // (0)(l — e~ 2a ~) 


(85) 
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From this equation, together with (78), we have 

r+°° 

?*■'=/ 


'+°° _ 1 

( 0 a,e +asin(y)) ,_ e 


_ V 2 

2c?dy 


r+ 


_ v 2 

/ 9 9 \ e 2o^ 

g (b 2 a + 0(a ) + asm(y)) _ dy 

Vino 

“h 00 qN (r\ 

——— [b 2 a 2 + 0(a 3 ) + asin(y))“ 

3 


+ ^ ^ (b 2 a 2 + 0(a 3 ) + asin(y))' 

+0 ((b 2 a 2 + 0(a 3 )+asin(y)) 4 ) 
a 2 g"(0) f +oa . 2 1 -J>L 

-o- / Sm ( }, ) ^(j g 2^Jj + 0(a 


v/_v 

,3\ 


= c "( 0 )( i - e -^) a2 + 

4 

Thus the equilibrium of the average system 0 is 


- ga,e - 

a,e 


f ^(0)( 3 - 4 ^ 2 + e ;- 2 ) fl 2 + o(fl 3)l 

24e"(0)(l-e- 2cr ) V ’ 

0 

^a,e 


5 "(0)(l- e -- 2) fl 2 + o(fl 3) 


The Jacobian matrix of the average system (74) at the equilibrium (0 a,e , <^ a,e , £ a,e ) is 

i a = 

j r 


1 ep 0 

£7 a 21 1 - £Wi 0 

1 — £W2 


f f a 0 

fcJ r 3i u 


where 


/ a — 
J r2l — 


W l 


Vino 

w 2 

r3 ‘ _ o 


I a — 

•Ml — 


/*-)-oo 


/»-|-00 


( 0 a,e + asin(y)) sin(y)e 2 a 2 dy, 

_ V 2 

;'( 0 a ’ e + asin(y))e dy. 


Thus we have 


det(Xl — Jf) 

= (A — 1 + £w 2) ((A — 1)^ + £wi (A — 1) — E 2 pJj 2 ^j. 

With Taylor expansion and by calculating the integral, we get 


p-|~o° 


g' ( 0 a,e + asin(y)) sin(y)<? 2 ^dy 
= aV2nog"(0) +0(a 2 ). 


( 86 ) 


(87) 


( 88 ) 


(89) 

(90) 


(91) 


(92) 
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By substituting ( [92] ) into ( |9lj ) we get 


det(Xl — 7 r a ) = (A — 1 +£W 2 ) ((A — l) 2 -f£wi(A — 1 ) 


£pw, V(o)(i-«- 2< ’ 2 ) 


e 2 pw\ 


0(a 2 


2 3 

= (A — i + £w 2 )(A — i — n i) (A — i — n 2 ). 


where Hi = £ 


-wi+Jw\+2pwiaq"{0){l-e 2a2 )+ 7 & 0 (« 2 ) 

'- - -2---, 11 2 = £ ~ 


-wi-sjwl+lpwxaq" (0)(1—e 2(y2 )+^^°( a2 ) 
2 


(93) 


. Since 


w 1 - 


g ( 0 ) < 0 , for sufficiently small a, \Jw\ + 2 pwia£ ,, ( 0 )(l — e 2cj2 )+ J^ a O{a 2 ) can be smaller than 
Thus there exist £j“ > 0, such that for £ G (0, £*), the eigenvalues of the Jacobian matrix of the average 


system (74) are in the unit ball, and thus the equilibrium of the average system is exponentially stable. 
Then according to Theorem [9} we have the following result for stochastic extremum seeking algorithm 
in Fig. [4[ 

Theorem 11: Consider the reduced system (68)-(69)-(70) under Assumption [5] Then there exists a 
constant a* > 0 such that for any 0 < a < a* there exist constants r £ > 0 , c e > 0 and 0 < y e < 1 such that 
for any initial condition |Aq| < r £ , and any d > 0, 


lim inf (k G N : |Af | > c e |Ao|(y e ) fc + = +°°, a.s. 

£—>-0 l J 


and 


limp{|Af|<c e |Ag|(y e ) fc + 5,VA = 0,l,...,[A/£]} = l, 

£->0 t J 


(94) 


(95) 


where Af 4 (0f,^',Cf)- 

natural number. 


g 111 (0){3-4e- 2 ^+e- 8a ~ 


24s"(0)(l-e- 


^ + OK), 0, 


g"(Q)(i-g' 


-2ct z 


a 2 + 0(a 3 ) ), and N is any 


These results imply that the norm of the error vector Af exponentially converges, both almost surely 
and in probability, to below an arbitrarily small residual value 8 over an arbitrary large time interval, 
which tends to infinity as the perturbation parameter £ goes to zero. In particular, the 0f-component of 
the error vector converges to below 8. To quantify the output convergence to the extremum, we define a 
stopping time 

T e 5 - inf jkGN: |Af| > c £ |Ag| (y £ ) k + s} . 


Then by (94) and the definition of Af, we know that lim r): = +°o, a.s. and for all k < T, 


£->0 


01 


r(0)( 3 -^ + g -V) 

24v"(0)(l — e~y) 


<c £ |A§|(k)‘ + S, 


(96) 
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which implies that 

| Q[\ — 0(a 2 ) +c e |Aq| (y £ ) k + 5, Wk < t £ . 

Since the nominal output y k — h(l(d* + d^ + a sin(v£ + i))) and (hoi)'(6*) =0, we have 

.,0 i.rua*\\ _ 


$-h(l(0*)) = 


- (+ a sin ( v k+ i)) :2 + O (( 8{ + a sin ( v k+ 1 )) ■ 3 ). 


Thus by ( [97] ), it holds that 

\y° k ~ h o l(d *)| < 0(a 2 ) + 9(5 2 ) + C £ |Ag| 2 ( y e ) 2k , Vfc < T e 5 , 
for some positive constant C £ . Similarly, by 


lim P {\y° k - h o 1(9*) | < 0(a 2 ) + 0(8 2 ) + C £ I Ag | 2 (y £ ) 2k , 


e -)>0 l 


Vfe = 0,1,, [A^/e]} = 1. 


2k 


(97) 


With the measurement noise considered, we obtain that 

|y t+1 -Ao/((?*)| < 0(a 2 ) +0(5 2 ) +c e |ASi 2 (r e ) 2i 
+ M, Vfc < rf, 

for some positive constant C £ , and moreover, 

limP{|y jt+ i-Ao/(0*)| < 0(a 2 ) + 0(8 2 ) +C £ \A £ 0 \ 2 (y £ ) 

+M, Vfe = 0,l,...,[iV/e]} = 1. 

Remark 5.1: For stochastic ES scheme for dynamical systems with output equilibrium map, we focus 
on the stability of the reduced system. Different from the deterministic ES case (periodic probing signal), 


the closed-loop system (63)-(66) has two perturbations (small parameter £ and stochastic perturbation 
{v*}) and thus generally, there is no equilibrium solution or periodic solution. So we can not analyze the 
solution property of the closed-loop system by general singular perturbation methods for both deterministic 
systems (IQ) and stochastic systems ( Ii28l0 . But for the reduced system (parameter estimation error system 
when the state is at its quasi-steady state value), we can analyze the solution property by our developed 
averaging theory to obtain the approximation to the maximum of output equilibrium map. 
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VI. Concluding remarks 

In this paper, we develop discrete-time stochastic averaging theory and apply it to analyze the con¬ 
vergence of our proposed stochastic discrete-time extremum seeking algorithms. Our results of stochastic 
averaging extend the existing discrete-time averaging theorems for globally Lipschitz systems to locally 
Lipschitz systems. Compared with other stochastic optimization methods, e.g., stochastic approximation, 
simulated annealing method and genetic algorithm, the convergence conditions of discrete-time stochastic 
extremum seeking algorithm are easier to verify and clearer. Compared with continuous-time stochastic 
extremum seeking, in the discrete-time case, we consider the bounded measurement noise. In our results, 
we can only prove the weaker convergence than the convergence with probability one of classical stochastic 
approximation. Better convergence of algorithms and improved algorithms are our future work directions. 
For dynamical systems, we only focus on the stability of parameter estimation error system at the 
quasi-steady state value (the reduced system). For the whole closed-loop system with extremum seeking 
controller, we will investigate the proper singular perturbation method in future work. 
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Appendix 

A. Proof of Lemma [7[ approximation in finite-time interval with continuous average system 
Fix T > 0 and define 

M'= sup |A c (t)|. (A.l) 

0 <t<T 

Since (X c (t),t > 0) is continuous and [0, T] is a compact set, we have that M' < +°°. Denote M — M' + 1. 
For any £ <E (0,£o), define a stopping time t £ by 

T e = inf{t >0: |A(t)| > M}. (A.2) 

By the definition of M (noting that |v| = |Xq = |X C (0)| < M 1 ), we know that 0 < T e < +°°. If T e < 4-°°, 
then by the definition of T e , we know that for any s < T e , |X(5 , )| < M. By Assumption 1, we know that 
there exists a positive constant Cm such that for any \x\ <M and any y, we have \f(x,y) < Cm- And thus 
by 0, we know that 

M < |X(t £ )| <M+eC m <M + £qC M - 


(A.3) 
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Denote M — M + £qCm- By Assumption 1 again, we know that there exists a positive constant Cy 7 such 
that for any \x\ < M and any y, we have \f(x,y) \ < Cy ; . It follows by Q that for any \x\ < M, \ f(x) \ < C^. 
From ([9]) and ( |TT[ ), we have that, for any t > 0 , 

X(t)-X c (t ) = A f(X(s),Y(e + s))-nx c (s))]ds 
Jo 

~ [ f{X(s),Y(£ + s))ds 

= [ t [nX(s)J(£ + s))-f(X c (s)J(£ + s))}ds 

Jo 

+ f t [nx c (s)J(e + s))-nx c (s))}ds- 
J o 

f f{X{s),Y{e + s))ds. (A.4) 

By Assumption 1 and the definition of /, there exists a positive constant Ky, such that for any xi,X 2 in 
the subset : = {x G M" : |jc| < M} of W 1 , and any y G M m , we have 


\f(xi,y)-f(x 2 ,y)\ <Km\xi-x 2 \, 
l/(*i) - /(* i) | < Km \xi -x 2 \. 


By (A.4), (A.51 and (A.6), we have that if t < T e A T, then 


Define 


\X(i)-X c (‘)\< k m f \X(s)-X c (s)\ds 

Jo 


+ 

A f(X c (s),Y(e + S ))-fm s ))}ds 
Jo 

+ 

[ f{X(s),Y(e + s))ds 



^ hn{t) 



(A.5) 

(A.6) 


(A.7) 


A r 


a(e) 

m 


\m-x c (t)\, 

sup [\f(X c (s)J(e + S ))-f(X c (s))]d S 

0 <t<T JO 


sup 

0<t<T £ AT 


f(X(s),Y(e + s))ds 


n(t) 


Then by <|A.7 ) and Gronwall’s inequality, we have 


sup A r < (a(e) + /3 

0<t<z £ AT 


(A.8) 
(A.9) 

(A. 10) 


< (a(e) +/3(e))e^ r . 


(A.ll) 
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Since for any t > 0, we have t — t m (, ) < e, and thus /3(e) < C^e. Hence 

limS(e) = 0. 

£-»0 


In the following, we prove that lim e ^o®(£) = 0 he. 


lim sup 

e ^°o<f<r 


[/(X c ( 5 ),y(e + 5))-/(X c (,))]^ 


= 0 a.s. 


For any h6 N, define a function X"^),,? > 0, by 


x"W=£x c (')/.t <s<s±1 ,. 


k =0 


Then for any n G N, we have 


(A.12) 


(A.13) 


(A. 14) 


sup |X 77 (s)|< sup \X c (s)\=M' <M. 

0<s<T 0 <s<T 


(A.15) 


By (A.5), (A.6), (A.14) and (A.15), we obtain that 


sup 

o<t<T 


= sup 

0<t<T 


[f(X c (s),Y(e + s))-f(X c (s))]ds 


{[nx c (s) : Y(e + s))-f(X’\s)J(e + s))} 


+[nx'\s),Y(£ + s))-f(X>\s))} 

+ [f(X>\s))-f(X c ( S ))}}ds\ 

< sup [ t \f(X c (s),Y(£ + s))-f(X n (s),Y(£ + s))\ds 
o<t<rJo 

+ sup f (f(X’\s) : Y(£ + s))-nX n (s)))ds 
0<t<T JO 

+ sup f \f{X n (s))-f(X c (s))\ds 
o<t<rJo 

<2K^T sup |X c (s)-X 77 (s)| 

0 <t<T 


+ sup 

0 <t<T 


f‘ ( nX'\s),Y(£ + s))-f(X'\s)))ds 


(A. 16) 
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Next, we focus on the second term on the right-hand side of (A. 16). We have 


sup 

o <t<T 


f‘ {nx'\s)J(e + s))-nx’\s)))ds 


— sup 

0 <t<T 


— sup 

0 <t<T 


— sup 

0<t<T 


'(j). Y(e + s)) - f(X"(s ))) Y. I,t <s< i±»,ds 

f n t — ^ n J 


k=0 


Jo A “ 0 V n n ) n 1 

I I, {fm-),Y(e + s))-f(X c (-)))d S 

k= 0 


»(W+1) 

< sup V 

0<t<T k=0 


' r.At 


, (*+l) 


'r.At 


A t / k k 

f(X\-),Y{ £ + s))-f(X\-)))ds 


where [f] is the largest integer not greater than t. For fixed n and k with k < n([T] + 1), we have 


sup 

o <t<T 


,*+1 


l At 


At / A _ k 

nx c (-),Y(e + s))-f(X c (-)))ds 


< sup 

0 <t<T 


k +1 


10 


At A k k 

f(X\ K -) 1 Y{ £ + s))-f(X*C-)))ds 


+ 


l At A k - k 

f(X c {-),Y( £ + S ))-f(X c (-)))d S 


= 2 sup 


o <t<^r 


C^m(t) f k — Jc 

f(X C (-),Y( £ + s ))~f(X C (-)) jds 


to 


n 


n 


+ f (f(X c d),Y(e+s))-f(X c ( k -)))ds 

■Jtmit) V n n J 

<2 sup [ tm{,) (f(X c (-),Y( £ + S ))-f(X c (-)))ds 


k+\ 


0<t< „ 


to 


n 


n 


+ 4C M e. 


(A.17) 


(A.18) 
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For the second term on the right-hand side of (A. 18), we have 


fhnit) { _ k _ _ k \ 

/ /(x c (-),y(£+,))-/(x c (-)) U 

Jo \ n n J 

[ ? / £ ]-! / Jr b 

= e £ /(X'(-),K(i+l))-/(V(-)) 

i =0 V n n 

i [ f / £ ]— 1 / b - k 

= £[»/«]777T7 I /(X'(-),r(i+l))-/(X'(-)) 

,to V " » 

/ 1 [ f / £ M £ k 

= e[t/e] ?—t £ /(Z C (-),F(/+1))-/(Z c (-)) 

V F/ e J /=0 77 77 


(A. 19) 


Then by (|A. 19), the Birkhoff’s ergodic theorem and [13, Problem 5.3.2], we obtain that 


lim sup 

e ^ 0 0<K*+i 


f" t1] (f(F( k ),Y(e+s))-f(X‘(-)))ds 
lo V n n 

— 0 a.s., 


(A.20) 


which together with (A.17) and (A.18) implies that for any n e N, 


lim sup 

e ^°o<r<r 


(f(X n (s)J(e + s))-f(X n (s)))ds 


= 0 a.s., 


(A.21) 


Thus by (A.16), (A.21) and 


lim sup |X C (5) — A" ( 5 ) | = 0, 

n ^°°0<s<T 


we obtain lim„^oosup 0 < r < r |/o[/(A c (>),y(£ + s)) -/(X c (»)] ds 


0 a.s., i.e. 


lima(£)=0 a.s. 

e^O 


By ( |A.8| ), ( |A.11[ ), ( |A.12[ ) and ( ]A.23[ ), we have 


(A.22) 


(A.23) 


limsup sup \X(t) — X c (t)| = 0 a.s. 

e —>0 0<t<T £ AT 


By (A.l) and (A.24), we have 


(A.24) 


limsup sup |X(f)| 

£->0 0<t<T £ /\T 

x c (t) | 

< limsup sup \X(t)-X c (t)\-\-M' 

£-»0 0<t<T £ AT 

— M 1 < M a.s. 


< limsup ( sup |X(t) — A c (t)| + sup | 
£^0 \0<t<T £ AT 0<t<T £ AT 


(A.25) 
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By (A.3) and (A.25), we obtain that, for almost every co E Q., there exists an £o(<u) such that for any 
0 < £ < £o(ft>), 

Te(co) > T. (A.26) 


Thus by (A.24) and (A.26), we obtain that 


limsup sup \X(t)—X c (t)\—0 a.s. 

£->0 0<t<T 


Hence ( |T6l ) holds. The proof is completed. 


(A.27) 


B. Proof of Lemma [2]' approximation for finite-time interval with discrete average system 
By Lemma 1, we need only to prove that 

lim sup |A d (Y) — A c (t)| = 0. (A.28) 

e ^°0 <t<T 

Let M' be defined in the above proof of Lemma 1. For any £ 6 (0, £o), define a time 
T d by 

r d = inf{t > 0 : |A d (t) | > M}. (A.29) 

By the definition of M (noting that |jc| = |A d | = |X C (0)| < M'), we know that 0 < T d < +°o. If T d < +oo, 
then by the definition of T d , we know that for any s < T d , |X d (s)| <M. By g, we know that 

M< |A d (r d )| <M + £C m <M + £oC m . (A.30) 

Noting that if t < T d A T, then 

\X c (t)\<M, \X d (t)\<M, (A.31) 

and for any xi,JC 2 in the subset D^ := |jc| < M} of M", we have 

l/0i)-/02)| <Km\x 1 -x 2 \ and |/(*i)| < C^. (A.32) 


By ( [TO] ) and ( [IT] ), we have 


rt pt 

/Y vd/^\\ T/vc/ n \\\T n / r(v d/ 


X\t)-X c (t)= (. f(x\s))-nx c (s)))ds - / f(X\s))ds. 

J® ^ f m(0 


(A.33) 


Then by (A.31)-(A.33) and the fact that t — t m ^ < £, we obtain that for any 0 < t < A T, 


|* d (t) -X c (t)| < Km / |A a (,)) -A c (,))|^ + C^£. 
Jo 


(A.34) 
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By (A.34) and the Gronwall’s inequality, we get 


which implies that 


sup |X d 0)-l c (»| <C^eexp(^r), 

0<t<x£AT 


(A.35) 


lim sup |X d (t) —X c (t)| = 0. 

£ ^°0<t<Z^AT 


By (A.l) and (A.36), we have 


lim sup sup |X d (t)| 

e->0 o< t <r^AT 

<limsup| sup |A d (t) — A c (t)| + sup |X c (t)| 

e^o \o<t<r*AT 0<t<x$AT 

< lim sup sup \X d (t) — X c (t) \ + M f 

e^O o<t<z?AT 

— M' < M. 


By (A.30) and (A.37), we obtain that, there exists an £q such that for any 0 < £ < £q, 


4 > t. 


(A.36) 


(A.37) 


(A.38) 


Thus by (A.36) and (A.38), we obtain that 


lim sup sup |X d (t)-X c (t)| = 0. 
e-m o<t<r 


Hence (A.28) holds. The proof is completed. 


(A.39) 


C. Proof of approximation results (18) of Theorem^ approximation for any long time with continuous 
average system 


Now we prove that for any 8 > 0, 


liminfit > 0: \X(t) —X c (t)\ > 5} = +°° a.s. 
£->0 


(A.40) 


Define 


Q' — ^ (O : limsup sup \X(t, co) — X c (t)| = 0, VT gN[, 
e^o o<f<r 


(A.41) 


where X(t,G))(— X(t)) only makes the dependence on the sample clear. Then by Lemma [TJ we have 

P(Q!) = 1. (A.42) 


Let 8 > 0. For £ e (0,£o), define a stopping time t| by 

Tg = infjt > 0 : |A(t)-A c (t)| > 8}. 


(A.43) 
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By the fact that Xo — Xo = 0, and the right continuity of the sample paths of (X(/) — X c (t),t > 0), we 
know that 0 < xjj < +°°, and if xf < +°°, then 

|X(Tg ) —X c (Tg )| > <5. (A.44) 


For any (0 G £l', by (A.41) and (A.44), we get that for any feN, there exists an £o(co, 8 ,T) > 0 such 
that for any 0 < £ < £q(( 0 , 8 ,T), 




which implies that 


lim x f («) = +°°. 


£-s>0 


Thus it follows from (A.42) and (A.45) that 


The proof is completed. 


lim To = 4 - 0 ° a.s. 
£—>0 


(A.45) 


D. Proof of approximation results (19) of Theorem [j] 


The proof is similar to the proof (Appendixes C and D) of the continuous-time averaging results in [11311 
by replacing Xf and X t with X(/) and X c (/), respectively. The only difference lies in that X(/) —X c (t) is 
right continuous with respect to t, while both Xf and X t in [U5ll are continuous. 


E. Proof of Theorem [?]' the stability of the continuous-time version ([9]) of the original systems with the 
continuous average system 

Since the equilibrium X c (t) = 0 of continuous average system (|3]) is exponentially stable, there exist 
constants r > 0,c > 0 and y > 0 such that for any |jc| < r, 

|X c (t)| < c\x\e~ v , V? > 0. 


Thus for any 8 > 0, we have 

{|X(0| >c\x\e~^ + 8 } C {|X(t)-X c (t)| > 8 }, 
which together with Theorem [3] implies that 

lim infft > 0 : |X(r)| > c\x\e~^ 4- 5} 

e->0 

> lim infit > 0 : |X(f) — X c (t)| > 5} = +°o a.s. 
£-»0 


Hence ( |2TT ) holds. 
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Let T(e) be defined in Theorem [3} Thus limg^oTe = +°°- Since the equilibrium X c (t ) = 0 of the 
average system is exponentially stable, there exist constants r > 0, c > 0, and y > 0 such that for any 
be I < r, 


|X c (t)| < c\x\e ^, Vt > 0. 


(A.46) 


Thus for any 8 > 0, we have that for any |jc| <r. 


I 


sup {\X(t)\—c\x\e > 8 

{0<t<T{e) 

= U {\ x ( t )\- c \ x \ e ~ v > 

0<t<T(e) 

c U {|X(,)-X'(,)|>5} 

0 <t<T(e) 

= { sup |X(0-X c (t)|>5[,, 
[0 <t<T(e) 

which together with result (|T9|) of Theorem [3] gives that 


limsup P< sup {\X(t)\-c\x\e > 8 

£—>0 [o<f<r(e) 


<limP< sup |X(t) — X c (t)\ > 8 )■ = 0. 
e_ >° [o<?<r(e) 


Hence (22) holds. The proof is completed. 


(A.47) 


(A.48) 


F. Proof of Theorem [5] 

By using Lemma [2j we can prove this theorem by following the proof of Theorem [3] We omit the 
details. 


G. Proof of Theorem [6] 

By using Theorem [5] we can prove this theorem by following the proof of Theorem [4} We omit the 
details. 


H. Proof of Lemma [7] 

By Lemma [2] and the time scale transform, we get 

limsup sup X%\ 

e-+0 0 <k<[N/e\ 

— limsup sup \X(ek) — A d (efc)| 

e^0 0<k<[N/e] 

= limsup sup |A(t) — X d (t)| = 0 a.s. (A.49) 

e —>0 0 <t<N 


Hence (28) holds. The proof is completed. 
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I. Proof of Theorem [S] 

(i) Noticing that [N/e] > N for £ < 1. Then by Lemma [7J we know that for any natural number N, 


lim sup \Xj c — Xjf\ = 0 a.s. 

e ^°0 <k<N 


(A.50) 


By ( A.50| ) and following the proof of Theorem [3j i), we can prove (29). 

(ii) By Lemma |7j we know that for any natural number N, sup 0</ , </v \X k —X^\ converges to 0 a.s., and 
thus it converges to 0 in probability, i.e. (f30|) holds. 


J. Proof of Theorem [9] 

By using Theorem [8] we can prove this theorem by following the proof of Theorem [4} We omit the 
details. 
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